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The Space Shuttle Integrated Main Propulsion System (IMPS) consists of the External 
Tank (ET), Orbiter Main Propulsion System (MPS), and Space Shuttle Main Engines 
(SSMEs). The IMPS is tasked with the storage, conditioning, distribution, and combustion of 
cryogenic liquid hydrogen (LH2) and liquid oxygen (L02) propellants to provide first and 
second stage thrust for achieving orbital velocity. The design, certification, and operation of 
the associated IMPS hardware have produced many lessons learned over the course of the 
Space Shuttle Program (SSP). A subset of these items will be discussed in this paper for 
consideration when designing, building, and operating future spacecraft propulsion systems. 
This paper will focus on lessons learned related to Orbiter MPS and is the first of a planned 
series to address the subject matter. 


I. Introduction 

T he Space Shuttle Integrated Main Propulsion System (IMPS) consists of the External Tank (ET), Orbiter Main 
Propulsion System (MPS), and Space Shuttle Main Engines (SSMEs). Extensive Ground Support Equipment 
(GSE) also exists to service, monitor, and maintain these elements. Overall, the IMPS is tasked with storage, 
conditioning, distribution, and combustion of cryogenic liquid hydrogen (LH2) and liquid oxygen (L02) propellants 
to provide first and second stage thrust for achieving orbital velocity. An overview of the Orbiter MPS portion of the 
IMPS is shown in Figure 1. As indicated in this figure, nearly all of the MPS hardware is located in the Orbiter aft 
compartment. This includes the LH2 system, L02 system, gaseous hydrogen (GH2) system, gaseous oxygen (G02) 
system, gaseous helium (GHe) system, gaseous nitrogen (GN2) system, and other miscellaneous items. A functional 
description and schematics of the MPS hardware are shown in Table 1 and Figures 2though4, respectively. 
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Figure 1: Orbiter MPS Hardware Located in the Orbiter Aft Compartment (LH2 in Red, L02 in Yellow) 


Table 1: Summary of Orbiter MPS Hardware 


Item 

Basic Function 

Approximate 
Hardware Dry 
Weight (lbm) 

LH2 System 

Pre-launch ET loading and propellant recirculation/conditioning, ascent 
propellant feed to SSMEs, and on-orbit propellant dump/vacuum inert for 
system safing 

2850 

L02 System 

Pre-launch ET loading and propellant bleed/conditioning, ascent 
propellant feed to SSMEs, and on-orbit propellant dump/vacuum inert for 
system safing 

2450 

GH2 System 

Pre-launch GHe pre-pressurization and ascent GH2 re-pressurization of 
ET ullage with GH2 from SSMEs via 3 active flow control valves (FCVs) 

100 

G02 System 

Pre-launch GHe pre-pressurization and ascent G02 re-pressurization of 
ET ullage with G02 from SSMEs via 3 fixed-orifice FCVs 

80 

GHe System 

Pre-launch, ascent, and shutdown purging of SSMEprebumers and 
seals, backup pneumatic actuation for select SSME cryogenic propellant 
valves, entry repressurization/purging of MPS lines /manifolds and other 
Orbiter compartments, and primary pneumatic actuation of all MPS 
cryogenic propellant valves 

1950 

GN2 System 

Pre-launch SSMEL02 dome purge using ground supplied GN2 

15 

Miscellaneous 

Point sensor box for ET propellant level sensors and engine cutoff (ECO) 
sensors, ullage presure signal conditioner (UPSC) for control of G02/GH2 
FCVs, dome heat shields for SSMEs, etc. 

675 


Total MPS Hardware Dry Weight = 

8120 
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Figure 2: LH2 and L02 Systems 



Figure 3: GH2 and G02 Systems 



Figure 4: GHe System 
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The Space Shuttle Program (SSP) can be divided into distinct eras marked by the following key milestones: start 
of detailed design and certification (July 1972), first flight (STS-1, April 1981), Challenger Accident (STS-51L, 
January 1986), Return to Flight following Challenger (STS-26, September 1988), first flight of Endeavour as the 
Challenger replacement vehicle (STS-49, May 1992), Columbia Accident (February 2003), Return to Flight 
following Columbia (STS-1 14, July 2005), and final flight (STS-135, currently planned for July 2011). Throughout 
these eras, the design, certification, and operation of the Orbiter MPS have produced many propulsion-related 
lessons learned. As the SSP comes to a close, an attempt is being been made to document these lessons learned and 
group them into a variety of common categories, as listed in Table 2, for future reference and use. Since a given 
lesson could conceivably belong to more than one category, a knowledge capture system has been established to 
allow primary, secondary, and tertiary categorizations if applicable. This knowledge base will be available to future 
NASA designers. 

Table 2: Orbiter MPS Lessons Learned Categories - Highlighted Items are Discussed in this Paper 


Requirements Development 

Project/Program Management 

Cost/Schedule Estimation 

Communication/Chain of Command 

Hardware Design (Vehicle/Sy stem/ Component) 

StructuraPThermaPFlow Analysis 

Analysis/Simulation (Vehicle/ System/ Component) 

Failure Modes and Effects Analy s is / Critial Items List 

Manufacturing/Assembly/Inspection 

Hazard Report Generation 

Component Development/Qualification Testing 

Reliability Analysis/Assessment 

Component Acceptance Testing 

NaturaPInduced Environments Definition 

Integrated System Development/Qualification Testing 

Materials and Processes (M&P) 

Integrated System Checkout/Acceptance Testing 

Component/System Cleanlines s 

Hardware Certification (Vehicle/System/Component) 

Fluid Cleanliness/Chemistry 

Ground Operations 

Configuration Control 

Flight Operations 

EEE Parts Selection/Testing 

Maintenance/ Sustaining Engineering 

Problem Report ing/Tracking/Clo sure 

Logistics Supportability 

Problem/ Anomaly Resolution 


It is not the intent to provide an exhaustive treatment of all Orbiter MPS anomalies and resolutions to date, nor to 
address Agency, program, or project management practices that may have influenced the design, construction, and 
operation of the Orbiter MPS over time. Instead, the items discussed herein are intended for consideration when 
designing, building, and operating future spacecraft propulsion systems. Most are equally applicable to single-use or 
reusable vehicles, although the significance could depend on which type of vehicle is being developed. Some are 
also very basic in nature, could apply to multiple subsystems/components, and would normally be considered in the 
context of engineering common sense. However, these principles can sometimes be neglected, either inadvertently 
or purposely, due to an incomplete understanding of relevant loads/environments, inadequate analysis/testing, 
inexperienced design/test personnel, or cost/schedule compromises that inevitably accompany spacecraft propulsion 
system development. 

Lessons learned from other categories listed above will be discussed in future papers. This first paper provides a 
summary of select lessons learned related to Orbiter MPS in the categories of hardware design and hardware 
certification, as highlighted in Table 2, spanning all eras of the SSP. Note that the term ‘hardware’ is used 
generically and could refer to the entire Orbiter MPS or to a specific subassembly or component within the Orbiter 
MPS depending on the context. 


II. Hardware Design Lessons Learned 

The Space Shuttle was originally developed for a variety of purposes: satellite delivery and retrieval, orbital 
servicing, round-trip service for science instruments, and laboratory research in space. The Orbiter MPS served a 
crucial role in enabling these objectives to be met. Based on the system description provided earlier, the Orbiter 
MPS contains the following types of hardware: tanks/accumulators, lines/manifolds/fittings/seals, flexhoses, valves, 
pressure regulators, disconnects/couplings, pumps, filters/screens, orifices, sensors/instrumentation (pressure, 
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temperature, liquid level, hazardous gas, etc.), control avionics, and secondary structure. Design-related lessons 
learned have been grouped in accordingly, although they are presented in no particular order. 


B. Valves and Regulators 

1. Topic: Oxygen Material Compatibility. 

Discussion/Observation(s): Early in the SSP, ET ullage pressure control during ascent required independent 
throttling of three, two-position flow control valves (FCVs) in the Orbiter MPS, each containing an internal bypass 
orifice in parallel with the main flow path. Warm (350 °F), high pressure (3000 psi) G02 tapped from the SSMEs 
was routed through the FCVs to the ET ullage for this purpose. The FCVs are shown schematically as part of the 
G02 system in Figure 3. Other details of the FCV are shown in Figure 5. 



Figure 5: Orbiter MPS G02 FCV Manifold and Valve Cross Section 

The FCV’s were subsequently redesigned to eliminate the bypass orifice due to performance problems, and the 
material of the poppet was changed from stainless steel to Inconel 718 after ignition occurred during aluminum 
particle impact testing at the NASA White Sands Test Facility (WSTF). The new redesigned FCVs shuttled between 
a low flow and a high flow position rather than between open and closed, and without the bypass flow path. 
Subsequent WSTF particle impact testing managed to ignite the Inconel 718 valve, however. 

Problem Resolution(s): The FCV poppet material was changed to Monel, which was shown by test to be much 
less sensitive to ignition by aluminum particle impact. In addition, the FCVs were eventually shimmed to a fixed, 
intermediate position suitable for all ET ullage pressurization in-flight operational and failure scenarios. 

Recommendation(s)/Lesson(s) Learned: There were many lessons learned as a result of this issue. First, the 
use of Inconel and Monel was determined to be preferable to stainless steel based on the reduced propensity to ignite 
upon particle impact. Second, testing should always be performed in a representative G02 environment to ensure 
that design sensitivities are identified early. Finally, careful consideration should be given to the use aluminum or 
aluminum alloys for piece-parts that experience relative motion in a G02 or L02 system. Material transfer and 
particle generation can cause an oxygen ignition problem, either locally (small particles act as kindling that can 
ignite) or elsewhere in the system (larger particles can impact downstream components and ignite). 

2. Topic: Position Indicator Mounting Location 

Discussion/Observation(s): The Orbiter MPS L02 and LH2 systems each contain two fill and drain valves, one 
inboard and one outboard, as shown schematically in Figure 2. The fill and drain valves are driven by a GHe 
pneumatic actuator and are used for propellant loading during prelaunch and residual propellant dump/system 
vacuum inert while on-orbit. The position indicators in these valves, as with several other MPS valves, are mounted 
in the drive train upstream of the valve element. This can be seen in the valve drawings shown in Figure 6. During 
vehicle checkout in preparation for launch, a latent defect in the design of the valve actuation mechanism resulted in 
partial galling of two concentric sliding surfaces and breakage of a shaft key in its key way. Without the key, the 
actuator was free to travel and engage its position indicators without driving the valve element to change position. 
Fortunately, the mechanism seized completely, making the operators realize the valve had not translated fully. The 
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potential is there, however, for an unrecognized driver galling failure to fool the operators into clearing a failed 
valve for launch. Other MPS propellant valves share this generic design oversight. 

Problem Resolution(s): A redesign of the fill and drain valve driver mechanism and position indicator mounting 
configuration was never undertaken. However, a significant amount of testing was performed on a discrepant valve 
showing early signs of driver wear/galling to determine how many missions could be performed prior to complete 
seizure of the drive train. The resulting data became the basis for a high-sensitivity low pressure actuation test 
(LPAT) performed on all four fill and drain valves prior to every flight to screen for incipient driver galling failures. 
In addition, the fill and drain valves are preemptively disassembled and inspected for driver wear every 16 flights 
during interval overhaul sessions. 

Recommendation(s)/Lesson(s) Learned: Valve position indicators should be installed downstream of the 
propellant valve element rather than upstream at the actuator or in the drive train. This approach was successfully 
implemented on the 17 inch L02 and LH2 disconnects, where the flapper position indicator is directly coupled to 
the flapper drive arm providing actual, not inferred, information on flapper position. 





Figure 6: Orbiter MPS L02 and LH2 Fill and Drain Valve Details 

3. Topic: Bellows Size and Manufacturing Process Control 

Discussion/Observation(s): Four configurations of two-way solenoid valves and two configurations of three- 
way solenoid valves are used throughout the Orbiter MPS to control the flow of regulated GHe to the SSMEs and 
provide pneumatic actuation pressure to the many L02/LH2 valves and disconnects located throughout the system. 
These valves are shown schematically in Figure 4. Other details are shown in Figure 7, including the presence of an 
electro formed bellows within each valve. 
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Figure 7: Two-Way Solenoid Valve (Left) and Three-Way Solenoid Valve (Right) 

Sensitivity to the manufacturing process was found during an investigation of squirmed bellows in the two-way 
and three-way solenoid valves. It was determined that the sub-tier vendor had changed a proprietary electroform 
nickel-cobalt plating process. Neither the prime contractor nor the valve vendor had adequate visibility into the key 
parameters associated with this process, including bath composition, plating temperature, bath flow characteristics, 
electrical charging of plating materials, etc. The change resulted in lot-to-lot variation in material properties and 
geometry of the bellows convolutes, as well as possible variation of those same characteristics within a given lot. 

Problem Resolution(s): The corrective action was to test many units and group the delivered bellows into three 
different lots, one of which was scrapped entirely due to inadequate performance. The earlier and later lots were 
found to have sufficient ductility to meet life cycle requirements and justify continued use in the valves. 

Recommendation(s)/Lesson(s) Learned: Small electroformed, edge-welded, or hydroformed bellows offer a 
way to provide internal sealing and achieve a desired force balance in valves without the weight penalty of strong 
springs or the wear and imperfect sealing characteristics of dynamic seals. However, these items are sensitive to the 
manufacturing process, and unrecognized changes in critical process parameters can lead to unacceptable variability 
and failure, particularly when subjected to a highly dynamic environment. As a result, their use should be minimized 
and, when they are necessary, the manufacturing process should be carefully documented and controlled. 

4. Topic: Fail-Safe Bellows 

Discussion/Observation(s): Seven pressure regulators are used in the Orbiter MPS to provide regulated GHe at 
750 psi to the SSMEs and pneumatic actuators on the many L02/LH2 valves and disconnects located throughout the 
system. These regulators are shown schematically in Figure 4. All share a common design with an edge- welded 
bellows used to provide internal sealing and force balance within the regulator. During testing of a GHe regulator at 
NASA Johnson Space Center (JSC), the bellows developed a fatigue crack resulting in significant internal leakage. 
The resulting force imbalance led the regulator to fail in a wide open condition. Had this type of failure occurred 
early in ascent or during certain phases of entry, it would have likely resulted in catastrophic overpressurization of 
the aft compartment. 

Problem Resolution(s): The regulator was redesigned to utilize a vented, atmospheric referenced bellows. A 
simplified cross section of the original and redesigned GHe regulator is shown in Figure 8. 
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Figure 8: Orbiter MPS 750 psi GHe Regulator, Old Config (Left) and Redesigned Config (Right) 

Recommendation(s)/Lesson(s) Learned: If bellows must be used in a highly dynamic component or system, a 
fail-safe mode should be designed-in to avoid an undesirable condition in the event of a bellows failure. 

5. Topic: GHe Check Valve Poppet Jamming 

Discussion/Observation(s): Five different configurations of check valves are used in the Orbiter MPS GHe 
system, accounting for 31 check valves total. Of those, 16 units are located in the high pressure section of the system 
upstream of the GHe pressure regulators. The high pressure check valves are shown schematically in Figure 4 and a 
general cross section of that hardware is shown in Figure 9. 



FLOW 


Figure 9: Generic Configuration of GHe Check Valves 

During ground testing, a fail open condition occurred on several high pressure check valves, resulting in 
excessive reverse leakge. Failure analysis efforts, including x-rays of the discrepant check valves prior to 
disassembly and inspection, revealed significant wear and contamination, but cocking of the poppet within the end 
piece was determined to be the root cause. 

Problem Resolution(s): The poppet was redesigned to provide a higher length-to-diameter (L/D) ratio, thus 
limiting the potential for cocking within the bore of the end piece during rapidly changing GHe flow demands (e.g. 
GHe regulator slam start). In addition, a new material was chosen for the poppet and end piece. In particular, the 
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poppet material was changed from 316 stainless steel to a more ductile and lubricious aluminum-bronze alloy to 
decrease localized friction and the potential for poppet binding/cocking. Finally, an in-flight checkout (i.e. pressure 
decay test) was added to evaluate reverse leakage of the check valves every flight prior to entry. 

Recommendation(s)/Lesson(s) Learned: Careful attention should be paid to the L/D ratio, radial clearances, 
and material selection when designing a sliding interface between valve piece-parts. Also, representative testing, 
including high flow-demand events and total commodity throughput, should be conducted early to confirm adequacy 
of the hardware design. 

6. Topic: Sharp Edges and Corners on Component Piece-Parts 

Observation(s): Many otherwise good component designs are hampered by the existence of sharp edges on 
sliding piece parts or only slightly broken edges leading to eventual binding/jamming and galling. Drawing callouts 
are for a maximum radius and “break edge” callouts are sometimes ignored. Excessively sharp corners on internal 
radii can also be problematic, resulting in significant stress concentration and fatigue cracking in service. 

Several notable failures involving a variety of Orbiter MPS hardware shown schematically in Figures 2through 4 
have resulted from these types of circumstances. A stress concentration feature on the L02 fill and drain valve’s 
needle bearings caused the tips to break off due to fatigue, leading to failure of the entire bearing assembly and 
release of debris into the system during Main Propulsion Test Article (MPT A) testing. A sharp edge left at the 
housing bore undercut on the GHe two-way solenoid valve, combined with a failure to inspect and deburr the 
undercut machining, resulted in the solenoid plunger catching on this sharp edge and creating more burrs, causing it 
to bind. Finally, sharp corners in rectangular through-holes on a GHe check valve poppet caused fatigue cracking in 
service and complete poppet head failure/liberation on a unit being flow tested at NASA JSC. 

Problem Resolution(s): For the fill and drain valves, the needle bearing assembly was redesigned to eliminate 
the stress concentration on the bearings and incorporate a locking feature to keep the bearing assembly from coming 
apart. For the GHe solenoid valve, design and inspection changes to implemented to eliminate detrimental sharp 
edges. For the GHe check valves, the poppet design was reviewed to ensure a generous radius had been specified 
and all units in service were inspected to confirm compliance with drawing requirements. Discrepant check valves 
were eventually replaced with spare units. 

Recommendation(s)/Lesson(s) Learned: Use generous radii at edges along sliding surfaces, at neck-downs, 
and on internal corners to prevent binding/jamming and fatigue cracking from localized stress concentration. 

7. Topic: Prevalve Detent Roller Cracks 

Observation(s): The Orbiter MPS uses six pneumatically actuated cryogenic prevalves, three each in the L02 
and LH2 systems, to control the flow of propellants to the SSMEs. These valves are shown schematically in Figure 
2. Due to the criticality of keeping the prevalves open in the highly dynamic environment of ascent, two spring- 
loaded detent rollers are used on each prevalve to supplement the GHe pneumatic actuation pressure. These 
cylindrical rollers, made of Vespel SP-21, are pressed into matching grooves in the pre valve visor, made of cast 
Inconel 718, using a set of Belleville springs. A drawing and several photos of the prevalve hardware are shown in 
Figure 10. 
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Figure 10: Orbiter MPS L02 and LH2 Prevalve Detent Roller Configuration 

Uneven wear and overload cracks were discovered on several occasions in the same roller location on one of the 
L02 prevalves in service. This was accompanied by additional witness marks and wear on other interfacing piece- 
parts within the detent roller stack. Initially, the investigation focused on a rough cast surface and excessively sharp 
edge radius on the detent groove in the affected visor. Howvever, the detent roller cracks recurred despite sanding of 
the contact surface on the visor and blending of the edge radius on the detent groove to meet drawing requirements. 

Another focus of the investigation involved a small dimple (i.e. blind hole) that was machined into the end of 
certain detent rollers. The purpose of this dimple was to provide a positive visual indicator of rollers manufactured 
from Kel-F material that had been successfully batch tested to confirm L02/G02 compatibility. Although this 
dimple was thought to be located in an area of low stress, the overload cracks initiated and passed through this 
dimple whenever present. However, one roller cracked without the dimple so it was not considered the root cause. 

Problem Resolution(s): Periodic inspections were instituted to regularly monitor the integrity of the affected 
L02 prevalve detent roller. In addition, all rollers with a dimple were purged from inventory and not permitted to be 
installed on the prevalves due to the apparent stress concentration effect. Finally, a solid modeling effort and 
detailed tolerance stackup analysis were performed. This revealed that the drawing tolerances would allow the 
Belleville spring stack to reach solid height under certain circumstances, thus putting excessive load on the detent 
roller. Once this possibility was recognized, parts of the affected detent roller stack were replaced by strategically 
chosen spares to avoid the tolerance stackup issue. Note that the original prevalve design drawings were never 
updated to change the tolerances that led to the solid height condition because of adequate field performance of the 
other hardware. 

Recommendation(s)/Lesson(s) Learned: A thorough tolerance stackup analysis should be conducted on all 
components containing moving parts to ensure that worst-case tolerances will not cause hardware malfunction or 
premature wear/failure. 

8. Topic: Design Control Preferred Over Operational Control 

Discussion/Observation(s): The L02 and LH2 prevalve configuration was discussed in the prior section. 
During LH2 system decay checks on the vehicle, excessive internal leakage caused by a broken main seal was 
discovered on one of the LH2 prevalves. It was determined that this condition was created by inadvertently 
slamming the visor open and closed during planned ground checkout. Normally, the valve was operated in a 
‘snubbed’ mode during cycling. This meant that GHe pneumatic pressure was applied to both sides of the actuator 
piston prior to venting pressure from the back side and allowing the piston to move. In this case, the prescribed 
operational sequence was not followed, driving the valve mechanism much faster than normal and damaging the 
main seal against its seat. 
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Problem Resolution(s): In addition to clarifying the operational procedures, a significant hardware modification 
was made to add open and closed anti- slam valves and plumb pneumatic GHe to the back side of the actuator piston 
whenever the front side of the actuator piston is pressurized. This was done for both the LH2 and L02 prevalves 
(and also the LH2 and L02 fill and drain valves). Orifices in the anti-slam valves are used to control the flow of 
GHe to the back side of the piston, while still allowing the L02 prevalves to meet their critical procurement 
specification requirement for closure within approximately one second of command at Main Engine Cutoff 
(MECO). This anti-slam mechanism, shown schematically in Figure 11, is now used in combination with the 
‘snubbing’ procedures used previously. 

Recommendation(s)/Lesson(s) Learned: Designed-in controls are always preferred over operational controls. 
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Figure 11: Orbiter MPS LH2 and L02 Prevalve Actuator with Anti-Slam Mechanism 

9. Topic: Porosity in Valve Castings as a Cause of External Leakage 

Discussion/Observation(s): The Orbiter MPS L02 and LH2 systems contain five different configurations of 
pneumatically actuated cryogenic ball valves. These valves are shown schematically in Figure 2 as PV13 through 
PV22. One of the five configurations is used for the LH2 topping and recirculation pump valves that are exercised 
during propellant loading/conditioning prior to launch. A diagram of this valve is shown in Figure 12. The LH2 
topping valve was removed after vehicle level leak checks found external leakage at one of the flange connections. 
Examination of the valve flange by the vendor revealed pitting on the sealing surface. Repeated attempts to 
eliminate the pitting via lapping of the flanges was unsuccessful due to porosity occurring throughout the entire 
flange depth. All ball valve housings made of cast A3 5 6 material were subject to this problem, as faster cooling of 
the molten material the flange region led to the formation of subsurface bubbles below the machining datum. It was 
determined that the discrepant unit had passed the acceptance test procedure (ATP) and previous post-installation 
leak checks because the seal had not landed on a surface pit at that time. 
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Figure 12: Orbiter MPS LH2 Topping and Recirculation Pump Valve 

Problem Resolution(s): Since the leakage depended on where the interface seal landed on the surface, the only 
option was to eliminate the porosity. The valve housings were changed to hot isostatic press (HIP) castings. The 
existing detailed inspection of both flange sealing surfaces was also enhanced in the ATP. 

Recommendation(s)/Lesson(s) Learned: In addition to surface finish and concentric lap requirements for 
cryogenic valves, consideration should be given to the casting process to help ensure that no surface or subsurface 
porosity exists. Design of a seal and gland area for which a repeatable footprint exists would also decrease the 
probability of erroneously satisfying a leak check due to the random seal / imperfection overlap. 

C. Tanks/Accumulators 

10. Topic: COPV: Porosity at Girth Weld 

Discussion/Observation(s): The Orbiter MPS GHe system contains ten spherical high pressure (4500 psi) 
Composite Overwrapped Pressure Vessels (COPVs), including seven small units (26 inch diameter, 4.7 ft 3 internal 
volume) and three large units (40 inch diameter, 17.3 ft 3 internal volume). These COPVs are shown schematically in 
Figure 4. The large units have a titanium liner and Kevlar/epoxy overwrap. The small units have either a titanium 
liner and Kevlar/epoxy overwrap (same as the large units) or a stainless steel liner and graphite/epoxy overwrap. A 
simplified diagram and a photo of the large and small GHe COPVs installed in a test stand at JSC are shown in 
Figure 13. 

Following a flight readiness test on the vehicle, excessive GHe leakage that saturated a hand-held mass 
spectrometer was found to be emanating from the body of one of the small COPVs, specifically the pneumatic GHe 
tank, TK4. Although the indicated leak rate was still within the allowable range, the affected COPV was removed 
for failure analysis. Similar GHe leakage issues were subsequently experienced on other COPVs so this was not an 
isolated occurrence. 
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Figure 13: Orbiter MPS GHe System COP Vs 

Problem Resolution(s): The failure analysis revealed numerous small cracks in the girth weld of the titanium 
liner, all of which were associated with weld porosity. Leak testing, radiographic inspection, and fractographic 
analysis confirmed that the observed leakage was associated with several of these cracks which had initiated around 
the weld pores during proof-sizing and grown through the liner thickness due to pressure cycling in service. Note 
that the affected COPV had passed a multiple radiographic and ultrasonic inspections, GHe leak tests, and a proof 
pressure test prior to delivery, and all weld pores were within acceptable limits for size and location/distribution 
given the liner material and wall thickness involved. Pressure cycle testing was also performed on the discrepant 
tank to determine the nature of how pores can lead to through-cracks from repeated high pressure cycles over time. 

Recommendation(s)/Lesson(s) Learned: External GHe leakage can manifest itself through the welds of a 
COPV even after a successful in-process inspections and final acceptance testing at the supplier. However, cracks 
originating from in-spec weld porosity will grow in a predictable and stable manner during pressure cycles and 
sustained high pressure application, resulting in detectable external leakage without compromising the structural 
integrity of the COPV. To minimize the potential for leakage, the maximum allowable pore size and 
location/distribution should be revisited. For tanks in service, the predictable growth characteristic makes every- 
flight leak checks a viable option. 

11. Topic: COPV Staged Pressurization 

Discussion/Observation(s): Following the Columbia accident, a review of the Orbiter COPV certification and 
operational practices commissioned by the NASA Engineering and Safety Center (NESC) raised concerns about 
stress rupture life predictions. Analysis indicated that the Orbiter COP Vs were closer to experiencing a stress 
rupture failure than was previously thought. It was also determined that the fleet leader COP Vs in test at JSC did not 
envelope the stress levels present in the Orbiter MPS GHe COP Vs. Stress Rupture life of a COPV is defined as “the 
minimum time during which the composite maintains structural integrity considering the combined effects of stress 
level(s), time at stress level(s), and associated environments.” Since a large portion of a COPV’s strength is derived 
from its fiber overwrap, the health of the fibers is crucial to preserving tank service life. External environments 
including humidity, temperature, vacuum, and ultra-violet light accelerate degradation of the Kevlar fibers, thus 
reducing the stress rupture life. A team of engineers and scientists from the academic, contractor, and civil service 
communities developed a method to calculate service life hours that would account for all of these complex 
variables. Tracking the equivalent hours that the flight COP Vs have accumulated was considered vital to managing 
risk for the SSP. 

Problem Resolution(s): The Orbiter MPS GHe COPVs are typically brought to a flight pressure of 4300-4400 
psia within 24 hours prior to launch. The analysis team found that temperature had a significant effect on COPV 
stress rupture life and a slight reduction in temperature could significantly reduce the impact on stress rupture life. 
Pressurizing the COPVs to the final flight condition in a single step caused the internal GHe temperature to rise to 
approximatley 190 °F. In order to mitigate the detrimental effects of temperature on stress rupture life, a staged 
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pressurization approach was adopted. Subsequent to that change, the COP Vs are first pressurized to 3600 psia while 
maintaining internal GHe temperatures below 140° F. After that initial pressurization, the internal temperatures are 
allowed to stabilize. During the final pressurization to 4435 psia, the internal GHe temperatures are not allowed to 
rise above 1 15°F. This operational change was considered to provide a significant benefit to the stress rupture life of 
the GHe COPVs. 

Recommendation(s)/Lesson(s) Learned: The effects of multiple pressure and temperature cycles over time 
must be applied cautiously when estimating the stress rupture life of a COPV. In the end, the stress rupture life of a 
COPV is defined by statistical treatment of empirically derived data on the fibers of the overwrap. In the future, 
multiple COPV burst tests combined with advanced aging tests of the fiber material should be used to gather all data 
relevant to stress rupture prior to certifying a COPV for use on a reusable or expendable flight vehicle. 

D. Lines/Manifolds/Fittings/Seals and Flexhoses 

12. Topic: LH2 Feedline Flowliner Cracks 

Discussion/Observation(s): The Orbiter MPS L02 and LH2 systems each contain three 12 inch diameter 
cryogenic feedlines that run from the 17 inch manifold to the inlet of each SSME. These feedlines are shown 
schematically in Figure 2 and their installation locations are shown in Figures 1 and 14. All 12 inch feedlines 
incorporate a gimbal joint with a bellows and flowliner near the SSME inlet. The purpose of the flowliner is to 
smooth flow over the gimbal joint bellows and extend bellows fatigue life by eliminating flow induced vibration 
(FIV) in such a dynamic fluid flow and acoustic environment. The flowliners included a slot pattern to facilitate 
cleaning and draining operations. A diagram of the LH2 feedline and a photo of a cracked gimbal joint flowliner are 
shown in Figure 15. 

In 2002, cracks were discovered in the Orbiter MPS 12” feedline flowliner slots, which are installed in the 
gimballing joint immediately upstream of the SSME low pressure pump inlets. The cracks were found to initiate 
around the slots, which had been punched (i.e. stamped, not machined) into the flowliner material (321 stainless 
steel on Columbia and the MPTA, Inconel 718 on all other vehicles) to facilitate cleaning and bellows inspection 
during feedline fabrication. The cracks were determined to be caused by previously uncharacterized vibro-acoustic 
coupling of the flowliner and slots with high order surging and rotating cavitation generated by the SSME low 
pressure fuel turbopumps (LPFTPs). 

Problem Resolution(s): All flowliner cracks were repaired by welding them closed while still on the vehicle. 
The flowliner slots were then precision-polished to remove any residual micro-cracks from the original slot 
punching process. Afterward, detailed inspections of all the slots using eddy-current and Repliset dental mold 
techniques were performed to baseline the condition of the hardware. Both inspection techniques were initially 
continued after every flight until mold contamination was discovered behind the flowliners. At that point, the mold 
impressions were discontinued in favor of eddy current plus visual inspection under magnification. 




12” LH2 
Feedline 
(3 PL) 
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Feedline and 
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Manifold 


Figure 14: Orbiter MPS L02 and LH2 Feedline Configuration (L02 on Left, LH2 on Right) 
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Figure 15: Orbiter MPS 12 inch LH2 Feedline and Gimbal Joint Flowliner with Cracks 

In addition to the in-place weld repair and inspection procedure development, a significant amount of testing and 
analysis was also performed to characterize the complex pressure/flow field, acoustic environment, and associated 
flowliner stresses/strains created by the LPFTP at the SSME inlet. This effort resulted in a placard being placed on 
the operational time of the SSME LPFTP at certain operating speeds believed to cause destructive vibro-acoustic 
coupling with the flowliner. This placard was tracked at the SSP integration level for the remainder of the program, 
and resulted in some SSME hardware swap-outs to protect the flowliners. 

Recommendation(s)/Lesson(s) Learned: Complex environments can couple with sensitive mechanical parts in 
fluid system components and cause fatigue cracking failures. As a result, the use of slotted flowliners in feedlines 
near engines should be avoided unless the manufacturing/inspection processes are chosen to minimize crack 
initiation sites, special care is taken to fully characterize the harsh environment at the engine interface to guide 
hardware design, and throrough testing is performed with representative hardware during certification (e.g. full 
range of pressures/temperatures, total operating time/propellant throughput, etc). 


13. Topic: Minimize Use of Flexhoses and Protect Them from Collateral Damage 

Observation(s): Six overbraided metal bellows flexhoses are used in the Orbiter MPS to transmit high pressure 
GHe for pneumatic actuation of the 17 inch L02/LH2 disconnects and low pressure GHe for 
purging/repressurization of the L02/LH2 manifolds. Flexhoses are needed for this purpose to accommodate the 
separation and ~2.5 inch retraction of the 17 inch L02/LH2 disconnects that join the Orbiter MPS to the ET. The 
flexhoses are shown schematically in Figures 2 and 4. Details of the flexhose installation and construction are shown 
in Figure 16. 





Figure 16: Orbiter MPS Flexhoses (3 on LH2 Side, 3 on L02 Side) and Construction Details 
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Due to recognized difficulties in achieving repeatable manufacturing processes, flexhoses are frequently 
overdesigned in terms of rupture pressure and cycle life capability. However, single event overload of a convoluted 
section, such as by stepping on the flexhose, can quickly reduce margins below minimum levels leading to 
premature failure in service. This type of collateral damage has occurred on numerous occasions due to high 
personnel traffic in the vicinity of the Orbiter/ET umbilical where the MPS flexhoses are located. In this area, where 
technicians routinely perform non-MPS pyrotechnic servicing and other electrical mate/demate work, the hoses have 
been used as a step or hand-hold by personnel unfamiliar with the sensitivity of this hardware. The results have 
included visible damage to the overbraid (i.e. “birdcaging”), external leakage of the hose, and unscheduled 
maintenance actions to replace damaged hoses. 

Problem Resolution(s): In response, the SSP imposed flexhose familiarization training for technicians and 
routine inspections for “bird-caging” of the external metallic overbraid, which could indicate potential overload of 
the underlying convoluted pressure carrier. Temporary covers were also made and installed during ground 
turnaround operations to avoid inadvertent personnel contact with the flexhoses. In some cases, the hoses were even 
removed or installed later than planned to minimize the potential for inadvertent personnel contact. 

Recommendation(s)/Lesson(s) Learned: For any spacecraft application, particularly on reusable vehicles, the 
use of flexhoses should be minimized. Where flexhoses are deemed necessary, they should located away from areas 
of high personnel traffic and protected from potential collateral damage. 

14. Topic: L02 Feedline BSTRA Ball Cracks 

Discussion/Observation(s): The Orbiter MPS 17 inch and 12 inch L02 and LH2 feedlines each incorporate two 
or three flexible ball strut tie rod assembly (BSTRA) joints to accommodate installation tolerances, relative motion, 
and induced axial loads during propellant loading, SSME firing, and Orbiter/ET umbilical retraction. The L02 and 
LH2 feedlines are shown schematically in Figure 2 and their installed locations are shown in Figures 1 and 14. The 
construction of each BSTRA joint includes two sets of tripod struts surrounding a central metallic ball and socket, 
with two bellows, one for the inner pressure carrier and one for the outer vacuum jacket, surrounding the entire joint. 
There are 18 BSTRA joints per vehicle that contain balls ranging in diameter from 1.25 inches to 2.24 inches. Note 
that all Orbiter balls are made of cast Stoody 2 material. 

During routine internal feedline inspections with a borescope, a cracked ball was observed in the manifold 
BSTRA joint of an installed 17 inch L02 feedline. Concerns associated with a cracked BSTRA ball included 
feedline structural failure (lack of load transmission through the ball causes a bellows failure), BSTRA joint 
malfunction (improper articulation due to friction/binding), and SSME damage/failure from the ingestion of foreign 
object debris (FOD) particles (ignition or structural failure due to impact, blockage of critical flow passages, etc.). 
Details of the L02 feedline layout, BSTRA joint construction, and cracked BSTRA ball are shown in Figure 17. 



L02 Feedlines BSTRAJoint Details 


Figure 17: Orbiter MPS L02 Feedline Configuration and BSTRA Joint Details 
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Problem Resolution(s): Due to difficulties in accessing the affected BSTRA joint, special visual inspection 
tooling had to be developed and demonstrated. Besides a borescope, this tooling included a pneumatically operated 
clamp to squeeze the struts together and loosen the ball within the cups, plus a long slender pole containing a 
manually driven wheel that could contact and rotate the ball. Rotating the ball was desired to assure full visual 
inspection of the surface through the narrow gap between the cups, but this was not possible in all cases due to 
binding of the ball in the as-installed feedline. Best effort inspections of the entire fleetconfirmed that only the 17 
inch L02 feeedline manifold BSTRA ball was cracked. As a result, the affected feedline was removed from OV-103 
and sent to the vendor for replacement of the discrepant BSTRA joint and extraction of the cracked ball for failure 
analysis. 

Failure investigation work determined that the BSTRA ball cracks were caused by a defect related to an 
inadequate manufacturing process and poor quality screening. Silica inclusions from the sand cast process remained 
on the ball surface and created a thermal expansion mismatch with the Stoody 2 parent material, resulting in the 
formation of small thumbnail cracks during high temperature exposure associated with application of the Vitro lube 
coating. The balls were dye penetrant inspected before that heat cycle because no such inspection was possible after 
Vitrolube application. Subsequent thermal stresses from cryogenic propellant exposure then propagated the cracks 
through the Vitrolube coating. 

A best-effort borescope inspection without ball rotation was later implemented on an every flight basis. In 
addition, a significant amount of testing was conducted on spare BSTRA balls to demonstrate crack arrest following 
thermal shock, characterize any FOD particles liberated from the cracks to ensure safe ingestion by the SSMEs, and 
verify continued functionality of a BSTRA joint with a cracked ball based on mechanical and thermal cycle testing. 
For new production BSTRA balls, a Hot Isostatic Press (HIP) process was used to reduce/eliminate subsurface 
voids. An improved ATP was also implemented, which included five cryogenic cycles in LN2 followed by a 
computed tomography (CT) and eddy current (EC) inspection before installation into the feed line. For the spare 
BSTRA balls in inventory, all units had the Vitrolube coating removed and underwent CT and EC inspection to 
screen out any units with existing crack indications. 

Recommendation(s)/Lesson(s) Learned: The casting process can cause voids/inclusions in the parent material, 
leading to thermally-induced crack formation during subsequent steps in the manufacturing process or while in 
service. The HIP manufacturing technique can reduce or eliminate voids in the cast material as it cools. In addition, 
an inadequate acceptance testing and inspection process failed to detect the small cracks. This screening process 
should be made as robust as possible. 

15. Topic: Torque relaxation in Cryogenic Joints 

Discussion/Observation(s): Numerous instances of loosened flange bolts at cryogenic joints in the LH2 system 
were discovered while investigating GH2 leakage detected by the Shuttle Hazardous Gas Detection System 
(HGDS). The affected joints had previously been assembled and passed a leak check with GHe at ambient 
temperature but subsequently began leaking GH2 at cryogenic temperatures. In general, when the interior of a line is 
chilled-down with cryogenic fluid, the bolted flange cools at a slower rate causing that area to remain at an elevated 
temperature for some period of time. This apparent thermal expansion of the bolted joint causes a temporary drop in 
preload. When the inside of the line is suddenly warmed, as when the system is inerted following ascent, the 
opposite effect occurs so the joint preload rises and may even yield the bolts slightly. This serves to reduce the joint 
preload at ambient temperature. Some of the affected flange joints used a Teflon-coated metallic seal while others 
used a fully non-metallic seal. With a non-metallic seal, it is possible for cold flow of the seal material to occur 
during the process, thus reducing the joint preload and sealing effectiveness permanently. This reduction may 
manifest at ambient temperature. 

Problem Resolution: A recurring post- flight torque check was considered for the bolts at all cryogenic flange 
joints in the LH2 system. However, this was not implemented due to complexities associated with access to each of 
the joints. Instead, only the affected joints were re-torqued and the risk of external GH2 leakage was mitigated via 
continued GHe leak checks at ambient temperature plus operation of the HGDS. 

Recommendation(s)/Lesson(s) Learned: Design cryogenic joints with materials having similar coefficients of 
thermal expansion. Consider geometry to minimize thermal gradients and relative thermal expansion. Design the 
lines and flange joint insulation for maximum access to bolts. Use Teflon-coated metallic seals rather than thicker 
non-metallic seals. If non-metallic seals must be used, allow time for cold flow of the material to occur, and perform 
a re-torque after 24 or 36 hours. 
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Miscellaneous / System Level 


16. Topic: Real-Time Haz-Gas Detection 

Discussion/Observation(s): The Space Shuttle hazardous gas (haz gas) detection system philosophy can be 
divided into two phases: pre-launch/loading and in-flight. The pre-launch monitoring system utilizes a high-fidelity 
mass spectrometer system, but gases must be pumped from the vehicle (e.g. orbiter aft compartment) into the 
Mobile Launch Platform (MLP) for analysis and measurement. In flight, haz gas detection is limited to six pyro- 
fired evacuated sample bottles in the Orbiter aft compartment, three of which are mounted in a rack by each of the 
two access doors in the Orbiter aft compartment. The bottles are fired individually at discrete points in time during 
ascent and the contents captured by each bottle can only be analyzed for excessive levels of of G02, GH2, and GHe 
during post- flight processing. A diagram and photo of the haz gas bottles is shown in Figure 18. 


LEFT SIDE SHOWN (RIGHT SIDE EQUIVALENT) 



AFT FUSELAGE 



Figure 18: Orbiter Aft Fuselage Gas Sampler System (OAFGSS) Haz Gas Bottles 

The haz gas philosophy described above created several dilemmas. First, the pre-launch time lag for transport 
and analysis of the sampled gases, combined with the relatively small number of sense port locations, could skew 
the concentration data with respect to time or spatial aspects (e.g., gas-mixing in the aft comparment was not well 
characterized), making troubleshooting difficult for suspected system leaks. Second, many fluid joints in the SSME 
are maintained at low pressure until engine start approximately six sec prior to liftoff, which means that leakage 
integrity of those joints can not be assessed with the high-fidelity pre-launch system due to the previously mentioned 
transport lag. Lastly, the pyro-fired sample bottles sometimes failed to operate correctly or had leakage issues that 
compromised the samples, and the few discrete data points measured in only two locations provided very little 
insight into the actual conditions in the aft compartment during ascent. 

Problem Resolution(s): The pyro valves on the haz gas bottles were redesigned to provide more effective 
sealing and reliable operation. The Orbiter Maintenance Requirements Specification (OMRS) was also changed to 
prevent post-flight SSME processing until the MPS Integration community had reviewed the in-flight haz gas bottle 
data. This was done to preserve the hardware state in case anomalies involving potential MPS system leakage were 
indicated by the data. In the post-Columbia accident time frame, an advanced haz gas monitoring sytem (AHGMS) 
was also proposed to continuously record pre-launch and ascent data using an array of nine “stick-on” solid state 
wireless haz gas sensors mounted throughout the aft compartment. A schematic representation of the proposed 
system is shown in Figure 19. 
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Figure 19: Proposed Advanced Haz Gas Monitoring System for the Orbiter Aft Compartment 

The sensors in this system were intended to be similar to those flown as a detailed test objective (DTO) on two 
Shuttle flights in the late 1990’s. The system would not have provided “real-time” data, but would have provided a 
fully continuous (i.e., not discrete) set of data points from a variety of locations within the aft compartment that 
could be downloaded post-MECO for immediate evaluation prior to entry. However, cost and schedule 
considerations prevented the proposed system from being implemented. 

Recommendation(s)/Lesson(s) Learned: The use of a real-time, on-board hazardous gas detection system 
would eliminate concerns about data time lag, gas mixing, and inability to evaluate all pressurized systems for 
potential leakage, and should be implemented on future hydrogen/oxygen space vehicles. 

1 7. Topic: Design for Proper Assembly 

Discussion/Observation(s): Several instances of incorrect assembly involving fluid and electrical system 
hardware have occurred on the flight vehicle and associated ground support equipment (GSE)/special test equipment 
(STE). Examples include improper connection of servicing commodities to fluid systems sharing a common 
disconnect/coupling design and service panel location, incorrect installation of fluid system components with 
common inlet/outlet tube stub sizes due to missing or ambigous flow direction markings, and improper assembly of 
electrical wire harnesses sharing a common connector design and adjacent location. Erroneous or ambiguous 
procedures and human error also played a role in some cases. Rework, and in some cases and hardware damage, 
could result from these types of issues. 

Problem Resolution(s): Various - generic hardware design guideline 

Recommendation(s)/Lesson(s) Learned: Preclude incorrect hardware installation/assembly by designing fluid 
and electrical system components and connectors to fit in one orientation/configuration only. Examples include 
physical separation, keyed electrical connectors and fluid disconnects/couplings, different size threads or tube stubs 
on the inlet and outlet of fluid system components, and clear flow direction markings on fluid system components. 
Other examples include design features that allow component piece-parts or seals to fit together in one way only, 
making further assembly impossible if not correctly positioned. The use of directional springs, shims, or spacers 
should also be minimized or eliminated where possible. The use of these design features/techniques will also aid in 
early identification of assembly errors via visual inspection and/or component or system level testing/checkout. 

18. Topic: Complications of Inerting Hydrogen Systems in Space 

Discussion/Observation(s): Following MECO, residual LH2 is expelled from the 17 inch feedline, 17 inch 
manifold, and 12 inch feedlines prior to cutting power to the SSME avionics and isolating MPS GHe pneumatics for 
the on-orbit stay. In the case of an early engine-out abort, residual LH2 must be expelled quickly to prevent re- 
entering the atmosphere with a hazardous quantity of propellant. During these events, residual LH2 will flash-freeze 
in the lines from exposure to the vacuum of space. For nominal on-orbit operations, a time delay of several minutes 
followed by a lengthy vacuum inert sequence allows the frozen H2 to sublime and be expelled through open valves 
over time. This software-controlled process is generically referred to the “MPS dump.” Early in the Shuttle Program, 
a crew switch throw was used to initiate the sequence. This early software sequence left a considerable amount of 
frozen H2 in the system, resulting in a significant buildup of pressure in the closed LH2 system. The resulting GH2 
then vented through the LH2 manifold relief valve’s outlet flame arrestor, causing heat damage to the vertical 
stabilizer, increasing the criticality of the relief system, and posing a risk to ground personnel post-landing. 
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Problem Resolution(s): The software was updated to greatly reduce post-dump residuals, thus eliminating 
unwanted cycling of the LH2 manifold relief valve on a routine basis. A later software update automated several 
features, eliminating the need for a crew switch throw and incorporating an automated response to system failures. 
Also, the entry point for the GHe used to help expel the residual LH2 from the manifold was changed from one 
location to another. 

Recommendation(s)/Lesson(s) Learned: The relief system should be a backup system by design, not one relied 
upon to prevent system or vehicle damage. In addition, it was learned that hydrogen tends to freeze on the cold 
walls of the system, making expulsion of the commodity difficult near its triple point. Relatively warm helium helps 
in sublimating the hydrogen, as does a lockup time before another period of opening the system to the vacuum of 
space. 

19. Topic: Effect of Instrumentation Resolution on Parameter Limits 

Observation(s): The Orbiter MPS contains a variety of sensors to monitor pressure, temperature, valve position, 
and voltage performance of the system during pre-launch and flight. The analog outputs from these sensors are 
routed through the Orbiter Instrumentation (01) system where Dedicated Signal Conditioners (DSCs) are used to 
generate equivalent digital signals for the purpose of onboard data recording and downlink via telemetry. The 
resulting minimum resolution for each measurement (also referred to as the ‘minimum bit’ or ‘bit toggle’) can vary, 
and issues can arise when comparing the digitized sensor outputs to established operational limits. For example, a 
set of launch commit criteria (LCC) or flight rule may be specified with a limit value that cannot be measured 
directly due to data resolution limitations in the 01 system. 

Problem Resolution(s): The SSP philosophy dictates that a conservative value be assumed when establishing a 
limit value in the LCC or flight rules. For example, if a redline value is 10 psi minimum but the bit toggle of the 
associated pressure sensor limits the ouput to a value of 8 or 12 psi, then the limit would be adjusted appropriately in 
a conservative direction. In this case, the operational limit would becomes 12 psia minimum. 

Recommendation(s): When defining the allowable limits for pressure, temperature, or other system parameters, 
the bit toggle of all associated sensors should be accounted for in establishing those limits. 

III. Lessons Learned in Hardware Certification 

Per SSP guidelines, not all components require formal certification for flight. In fact, it is emphasized that system 
level design verification is preferred over individual component certification wherever possible to reduce cost and 
schedule. This type of integrated approach can also produce a more realistic and comprehensive verification of the 
hardware. This basic philosophy led to integrated testing on the Main Propulsion Test Article (MPTA), GHe system 
test stand, Orbiter/ET umbilical, Orbiter/Ground T-0 umbilical, and Orbiter/ET integrated test stands. The results of 
these integrated tests provided improved confidence in hardware design and system integration, propellant loading 
procedures, contingency procedure development, and identification of weaknesses or areas of improvement. 

When confidence in the design cannot be satisfactorily established at a higher level of integrated assembly, a 
component’s design may be certified by analysis, similarity, qualification testing, or any combination of those three 
techniques. With the many unique and complex hardware designs used in reusable space vehicles, the dominant 
means for certification is often qualification testing of individual components, or analysis supplemented by testing. 

Qualification testing of individual components can be expensive, often resulting in compromises being made. 
For example, a reusable vehicle designed for 100 flights woud ideally require at least 100 thermal cycles on 
cryogenic components during qualification testing. Additional cycles would be needed if a scatter factor is applied to 
cover contingencies, analytical uncertainties, unit-to-unit variation, and test-setup peculiarities. For large cryogenic 
valves requiring chilldown to sub-cooled LH2 temperature, such thermal testing would be complex and take an 
extremely long time to complete, making it impractical and expensive. As a result, the vast majority of the MPS 
cryogenic valves were only tested to a maximum of three thermal cycles. A related compromise involved the use of 
liquid nitrogen (LN2) instead of sub-cooled LH2 for qualification testing. Cryogenic life cycle testing can be 
performed more quickly, safely, and inexpensively using LN2, but with a corresponding loss of fidelity in 
accomplishing the goal of qualification testing: uncovering design defects. 

Several notable deficiencies in the area of component certification have been identified over the years during 
problem investigations and dedicated certification reviews. A numer of these deficiencies are discussed in the 
sections that follow. 

A. Topic: Representative Vibration Testing 
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Observation/Discussion: Random vibration testing during flight certification is intended to encompass worst 
case (flight and ground) conditions. A combination of instrumented ground/flight testing and analysis may be 
needed to define the appropriate power spectral density curves in the applicable environmental test requirements 
document. Once the magnitude and duration of that environment has been properly established via translation to the 
component, the hardware must be tested using flight representative mounting hardware, fluid lines (including tie- 
downs), and electrical harnesses (including tie-downs) in order to provide an acceptable level of fidelity. 

Example(s): Improper mounting/restraint of the 3/8 inch long outlet tube on an installed GH2 flow control valve 
(FCV) led to fatigue cracking and external leakage of a seal weld that was inadvertently subjected to additional 
external loads resulting from component installation. This issue was discovered during mass spectrometer leak 
testing after the STS-41 aft compartment H2 concentration investigation. It was suspected that qualification life 
testing on the FCV was performed with an incorrect line clamping configuration, imparting a different and less 
severe set of loads into the outlet tube. Due to the potential for an underqualified condition, an every- flight leak 
check was implemented. This was considered adequate since there was good correlation between the ambient 
temperature GHe leak checks and ambient temperature GH2 leakage. In addition, analysis determined that a leak 
rate three orders of magnitude higher than observed on the vehicle would be needed to result in a flammable 
concentration of H2 within the aft compartment, assuming perfect mixing. 

B. Topic: Periodic Review of Environments for Impact on Certification 

Observation/Discussion: Following the Columbia accident, a Shuttle “Mid-life Certification Review” was 
initiated to determine if the vehicle was still being operated and flown within the original design certification limits. 
These reviews were conducted on each vehicle subsystem, including Orbiter MPS, and looked for instances where a 
performance or environmental parameter was not adequately considered during initial certification or where the 
required performance/environments had changed over time (“mission creep”). The resulting certification 
discrepancies were generically referred to as “poke-outs.” 

Example(s): Several poke-outs related to mission creep were identified. One example involved the number of 
pressure cycles per flight on the GHe 2-way solenoid valve, which greately exceed the predicted value at the time of 
original qualification. Another example involved the 750 psi GHe regulator which was certified to an initial set of 
flowrates, but SSP changes brought about the use of additional flowrates by the SSMEs. The 750 psi regulator was 
also certified for an inadequate number of slam starts (supplying inlet pressure quickly without a downstream flow 
demand). Changes in ground checkout requirements had driven the number of slam starts up but, fortunately, the 
actual number of slam starts was already being tracked due to a known limited life on the bellows. To address this 
shortcoming, a delta qualification test was performed to demonstrate the capability for additional slam starts. 
Similarly, the GHe accumulator was being pressure cycled at a great predicted to exceed its certification by a factor 
of four. This issue was addressed by performing fracture analysis to clear the hardware for 5,000 maximum pressure 
surge cycles. Numerous other poke-outs were found on components throughout the system, all of which were 
addressed via additional analysis, testing, hardware rework, or appropriate risk acceptance rationale. 

C. Topic: Cerification by similarity 

Observation/Discussion: Certifying a component design by similarity to one that has previously been 
qualification tested for a different application offers a variety of cost and schedule benefits. Per SSP guidelines, 
“Similarity analysis may be used in lieu of tests where it can be shown that the article is similar or identical in 
design, manufacturing process, and quality control to another article that has been previously certified to equivalent 
or more stringent criteria.” When certifying by similarity, it is critical to ensure that all attributes of the hardware 
and new application (e.g. materials, loads, fluid and environmental exposure environments, etc.) are truly similar to 
the prior application. Any differences or omissions must be addressed by supplemental analysis and/or delta 
qualification testing. Further, certification by similarity should only extend to a single design beyond the originally 
certified hardware. For example, certifying hardware by similarity should be avoided when the reference hardware 
was certified by similarity to yet another previous design. This is referred to as “daisy-chaining” and it could result 
in inadequate certification coverage. 

Example(s): One example of this principle involves the Orbiter MPS cryogenic pressure and temperature 
transducers. These items were certified for use on Apollo by similarity to earlier programs where testing was 
conducted in the late 1950’s and early 1960’s. As a result, a high failure rate of these transducers has been 
consistently observed over the years on Shuttle. Because there is redundancy in the transducer applications on 
Shuttle, no requalification effort was undertaken. 
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D. Topic: Testing in LH2 

Observation/Discussion: Careful consideration should be given to the often more expensive but much more 
realistic option of testing with the correct saturated cryogenic fluid (LH2). The downstream savings in safety and 
reliability may be well worth the initial cost and schedule impact. 

Example(s): One example involved a 2 inch cryogenic ball valve that jammed during LH2 life cycle testing after 
passing ATP in LN2. (Life cycle qualification testing was coincidentally occurring because of the need to certify a 
new ball valve vendor). Failure analysis identified an oversized piston seal groove as the cause, but this failure 
underscored the fact that the LN2 ATP was inadequate to screen for certain piece-part defects. As a result of this 
anomaly, the ATP was changed to require LH2 testing of the ball valves. If budget had been particularly tight, 
another option might have been to rely on enhanced piece-part inspections, keeping LN2 testing in place for gross 
detection of non-conformances in a given valve build. 

Another example involved the LH2 17 inch disconnect shaft seal leakage that was partly responsible for the 
STS-35 and STS-38 launch delays during the summer of 1990. Several instances of launch delays caused by H2 
leakage have occurred throughout the SSP, most notably on STS-7, STS-29, STS-45, STS-47, and STS-80. Special 
drag-on leak detectors during a tanking test, correlated with the propellant loading timeline, identified a leak at the 
ET-side flapper shaft and follower arm shaft seals. Acceptance testing of the ET half disconnects used LN2 instead 
of LH2 at the time. As the instrumented tanking tests showed, leakage values could increase greatly at sub-cooled 
LH2 conditions when compared to LN2 conditions. This illustrates the difficulty in pinpointing cryogenic leaks 
when testing with anything but sub-cooled LH2. In fact, a reliable correlation between leakage at LN2 vs. LH2 
temperatures cannot be readily established without detailed knowledge of the joint design, combined with relevant 
test history on that particular joint. Subsequently, the ATP was changed to require the use of LH2 for cryogenic 
acceptance testing. 

E. Topic: Demonstration of Margin During Qualification Testing 

Observation/Discussion: Designed-in redundant features or analytical margin are not always verified by test. 
Design margin for a pressurized component should be demonstrated during qualification testing by performing burst 
testing, verification of redundancy, and operation with off-nominal inputs. Demonstrate workmanship margin during 
acceptance testing where contamination or differential expansion of piece-parts may impede intended function. 

Example(s): One example of this concept involves the burst testing performed on propellant components and 
pressurized vessels. For Orbiter MPS hardware, common burst factors are 1.5, 2.0, and 4.0, depending on the 
component, which are applied near the end of qualification testing. The pass fail criterion is that the unit holds 
pressure for a given amount of time without permanent deformation/distortion. There is no requirement for 
performance after the burst test as it is only testing the analytical design margin for strength. 

Another example involves testing a valve with a single position retention feature in place where dual retention 
features are designed-in and normally present in service. This proves that a single feature can keep the valve open 
or closed under a given set of loads and environmental conditions. This also demonstrates that the second feature 
can be considered fully redundant to the first. This type of margin testing was performed post-Challenger with the 
pre valve. In that case, open position pneumatic pressure was removed, and the valve was flow tested in the open 
position to prove that it would not fail closed in flight even with a loss of internal redundancy. Flow testing was 
conducted with both detents or equivalent load cells in place and at 65% and 109% of rated flow. 

One feature that is commonly verified is the ability of a propellant valve to function properly with low GHe 
pneumatic supply pressure, such as 400 psi instead of the nominal 750 psi. When performed during ATP, such as 
with proof pressure testing and low pressure actuation testing at cryogenic temperatures, workmanship margin is 
demonstrated. 

F. Topic: Qualification Testing for Combined Environments 

Observation/Discussion: Spacecraft hardware experiences a combination of applied loads and environmental 
conditions, such as thermal, pressure, flow, and mechanical loads, both in flight and during pre-flight ground 
preparation. However, due to technical practicalities and cost/schedule constraints, only a few test conditions can 
reasonably be combined during qualification testing. 

A potential solution is to test more than one specimen, combining a subset of loads on one unit and another 
subset on a different unit, but this is not typically done due to cost and schedule constraints. Another potential 
solution is to test such environments serially, but since this can result in an overtest for the qualification hardware in 
some respects, a failure may not be truly indicative of a design deficiency. Another drawback of such serial testing is 
that it does not truly combine environments at the same time. 
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Whenever practical, loads should be combined during qualification testing. Future launch vehicle design should 
build upon experience gained during individual hardware certification methodology successes and failures. 

Example(s): As an example, a cryogenic valve may simultaneously experience very cold temperature, fluid 
forces on the closure device, installation and engine thrust loads, and random mechanical and flow-induced 
vibration, all while being required to cycle open and closed within given response time constraints and meet other 
critical performance requirements. Particularly for cryogenic valves, it is difficult to test under both vibration and 
fluid flow conditions. For larger valves, fluid flow testing is often omitted entirely due to cost, risk, and complexity. 
Where fluid flow testing was performed for Orbiter MPS components, it was often not done until a problem was 
encountered in service that drove the need for such a test. 

IV. Conclusion 

The design, certification, and operation of the Shuttle IMPS have produced many lessons learned over the course 
of the SSP. A subset of these items related to the Orbiter MPS were discussed in this manuscript for consideration 
when designing, building, and operating future spacecraft propulsion systems. The experiences and lessons learned 
discussed herein are chiefly applicable to main propulsion systems using L02 and LH2 propellants on reusable 
crewed spacecraft, but non-crewed or expendable launch vehicle propulsion systems could benefit from a review of 
the this material. Likewise, non-cryogenic propulsion systems could also benefit from a review of this material due 
to the inclusion of universal topics such as GHe pressurization components and tank-related issues. 

This manuscript divided discussion topics into two general categories: hardware design and hardware 
certification. Within the hardware design section, experiences and lessons learned were discussed in the following 
subject areas: valves and regulators, tanks/accumulators, lines/manifolds/ fittings/seals and flexhoses, and other 
system-level integration issues. Within the hardware certification section, experiences and lessons learned were 
discussed in the areas of environment testing and margin demonstration. 

Additional manuscripts involving Orbiter MPS lessons learned are planned for future publication. Examples of 
potential content include the following (in no particular order): system instrumentation configuration, probabilistic 
risk analysis (PRA), performance and failure trending, design for maintainability, requirements creep over time, 
aging vehicle / program issues, use of modified catalog parts, Orbiter/ET disconnect development and performance 
issues, document and data preservation, test stand maintenance, system cleanliness and contamination control, 
vacuum-jacketed feedline development and maintenance/repair issues, incremental system modifications, and many 
others. This incremental approach is thought to provide the most manageable and comprehensive means of 
documenting the many important experiences and lessons learned on Orbiter MPS. 
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Acronym List 


AHGMS 

Advanced Haz Gas Monitoring System 

ATP 

Acceptance Test Procedure 

BSTRA 

Ball-Strut Tie Rod Assembly 

COPV 

Composite Overwrapped Pressure Vessel 

CT 

Computed Tomography 

DSC 

Dedicated Signal Conditioner 

DTO 

Detailed Test Objective 

EC 

Eddy Current 

ECO 

Engine Cutoff 

EEE 

Electrical, Electronic, and Electromechanical 

ET 

External Tank 

FCV 

Flow Control Valve 

FIV 

Flow Induced Vibration 

FOD 

Foreign Object Debris 

GH2 

Gaseous Hydrogen 

GHe 

Gaseous Helium 

GN2 

Gaseous Nitrogen 

G02 

Gaseous Oxygen 

GSE 

Ground Support Equipment 

H2 

Hydrogen 

HGDS 

Hazardous Gas Detection System 

HIP 

Hot Isostatic Press 

IMPS 

Integrated Main Propulsion System 

JSC 

Johnson Space Center 

LCC 

Launch Commit Criteria 

L/D 

Length-to-Diameter ratio 

LH2 

Liquid Hydrogen 

LN2 

Liquid Nitrogen 

L02 

Liquid Oxygen 

LPAT 

Low Pressure Actuation Test 

LPFTP 

Low Pressure Fuel Turbopumps 

MECO 

Main Engine Cutoff 

MLP 

Mobile Launch Platform 

M&P 

Materials and Processes 

MPS 

Main Propulsion System 

MPT A 

Main Propulsion Test Article 

NASA 

National Aeronautics and Space Administration 

NESC 

NASA Engineering and Safety Center 

02 

Oxygen 

OAFGSS 

Orbiter Aft Fuselage Gas Sampler System 

01 

Orbiter Instrumentation 

OMRS 

Orbiter Maintenance Requirements Specification 

PRA 

Probabilistic Risk Analysis 

PSI 

Pounds per Square Inch 

SSME 

Space Shuttle Main Engine 

SSP 

Space Shuttle Program 

STE 

Special Test Equipment 

STS 

Space Transportation System 

T-0 

Time-zero 

UPSC 

Ullage Pressure Signal Conditioner 

WSTF 

White Sands Test Facility 
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